Predicting memory from the network structure of naturalistic events

When we remember events, we often do not only recall individual events, but also the connections between them. However, extant research has focused on how humans segment and remember discrete events from continuous input, with far less attention given to how the structure of connections between events impacts memory. Here we conduct a functional magnetic resonance imaging study in which participants watch and recall a series of realistic audiovisual narratives. By transforming narratives into networks of events, we demonstrate that more central events—those with stronger semantic or causal connections to other events—are better remembered. During encoding, central events evoke larger hippocampal event boundary responses associated with memory formation. During recall, high centrality is associated with stronger activation in cortical areas involved in episodic recollection, and more similar neural representations across individuals. Together, these results suggest that when humans encode and retrieve complex real-world experiences, the reliability and accessibility of memory representations is shaped by their location within a network of events.

event text descriptions. The Jaccard indices were computed within each annotator, and then averaged across annotators within each movie. As in the USE-based narrative networks, event centrality was defined as the normalized node degree. We found that the semantic centrality computed from the networks based on Jaccard indices was positively correlated with the semantic centrality based on USE embeddings (r(202) = .64, p < .001, 95% CI = [.55, .71]) and also with recall probability (r(202) = .27, p < .001, 95% CI = [.14, .39]). Second, we created networks whose edge weights between events were the cosine similarity between the word embeddings of the events. Specifically, the word embedding of an event was generated by averaging the word vectors (based on Google's pre-trained Word2Vec model; GoogleNewsvectors-negative300-SLIM) of unique words contained in the text description of the event, separately for each annotator. The word embeddings were then averaged across annotators.
Words that were not included in the pre-trained Google database were excluded from the analysis. The centrality (normalized node degree) computed from these networks was again positively correlated with the USE-based semantic centrality (r(202) = .54, p < .001, 95% CI = [.43, .63]) and with recall probability (r(202) = .34, p < .001, 95% CI = [.21, .46]). The camera pans into an animated scene with a teenage girl in a room in a tall building. The camera is inside the apartment, and the girl opens the pizza box and grabs a pizza.

9
The girl, before she can eat, hears a knock on the door. The girl opens the door and looks around to see no one is there.

7
The girl looks down and sees a package for her in an envelope at her doorstep. The girl goes back to her chair and to her pizza and opens the envelope. The girl pulls out a package with a disc inside it that says "A Single Life."

7
The girl pulls out the record disc from the package and the title "A film by: Job, Joris, and Marieke"

6
The girl gets up and puts the record disc into the record player and puts down the needle. The song on the disc plays and she sits down and begins to eat her pizza.

5
The girl is about to eat the pizza but then there is a flash. She stops and then the song plays its lyrics on the song. Part of the pizza is gone.

6
The girl realizes her power and gets up then lifts the needle on the record player. The disc plays the song and then the flash goes to her as a pregnant woman. The woman stops the record player and stops the song from playing.

4
The woman pulls the disc forward and back and sees the baby develop and devolve like the pizza before. The woman pulls the disc forward and the baby develops and pops into her arms but the baby starts crying.

4
The woman then flashes into her childhood self and looks at herself. The girl goes up to record player and tries to stop it but pops off the needle.

3
The girl flashes to her in a wheelchair as an elderly woman and she looks at herself. The woman rolls up her wheelchair but she flashes back to the same scene over and over again, getting frustrated. The woman rolls up again and flashes but then stops rolling up and finds that nothing happens. She then tries to roll really fast but falls back.

2
The woman gets up and she is an old woman who needs a walker and is wearing glasses. The woman sees that the song is about to end and tries to get to the record player.

8
The woman turns into ashes in a pot in the same nursery home and the record player stops with the needle lifting.

Supplementary Figure 1. Similarity of movie event descriptions across annotators. a.
Visualization of three independent annotators' movie event descriptions as trajectories in the Universal Sentence Encoder (USE) 2 text embedding space (for a related method, see ref. 3 ). Tdistributed stochastic neighbor embedding (t-SNE) was applied on the USE vectors (concatenated across annotators) for dimensionality reduction into a two-dimensional space. Events within each movie formed visible clusters in the space, and the overall configuration of the trajectories was highly similar across annotators. Each dot represents a movie event. Temporally adjacent events are connected with gray lines. Different colors indicate different movies. b. Two example movies' annotation trajectories from the three annotators (isolated from the trajectories in a). Numbers and the color of dots indicate the order of events within each movie. Dots (events) in brighter colors were presented earlier in the movie. c. Cosine similarity between the USE vectors of all 202 events (combined across 10 movies) generated from each annotator's movie event descriptions. Each black square on the diagonal indicates an individual movie (i.e., withinmovie similarities). d. We performed a randomization test (1000 iterations, one-tailed) to test the statistical significance of the cross-annotator similarity between movie event USE vectors. The red line shows the true mean event-wise cross-annotator cosine similarity between all possible annotator pairs. The histogram shows the null distribution of the mean cross-annotator similarity, generated by shuffling the event labels within each movie and annotator. The mean crossannotator similarity was significantly greater than zero (M = .78, p = .000999). Supplementary Figure 8. Cortical responses at between-movie boundaries during movie watching. a. Example movie frame images around a boundary between two movies presented in the movie watching phase of the fMRI experiment. At between-movie boundaries, the last scene of the preceding movie was followed by a 6-second-long title scene of the upcoming movie. The transition between the 39-s introductory cartoon (presented at the beginning of each scanning run) and the first movie of each scanning run was also counted as a between-movie boundary. b.

Supplementary
Whole-brain maps of z-scored cortical blood oxygenation level dependent (BOLD) signals from 10 TRs before to 29 TRs after between-movie boundaries during movie watching (TR = 1.5 s). The BOLD signals were averaged across times within each 10-TR time window and then across movies and participants. Time zero means the onset of the movie title scene. The maps were arbitrarily thresholded to visualize brain areas whose activation was relatively higher (red-yellow) or lower (cyan-blue) than the mean activation across all time points within a scanning run (z = 0). Between-movie boundaries evoked transient changes in activation across widespread cortical areas. The black outlines indicate the posterior medial cortex (PMC) and early visual cortex (EVC) regions-of-interest. c. Activation time courses around between-movie boundaries in PMC (left) and EVC (right). Gray lines show individual participants' time courses, averaged across all between-movie boundaries. Black lines show the averages across participants. The four shades of the gray bars at the top of each panel correspond to the four time windows used in b. Source data are provided as a Source Data file. d. Intersubject pattern correlation between the mean activation patterns of the first four events in each of the 10 movies. Each row and column of the similarity matrix represents an event, and the events are grouped by their temporal positions in the movie (i.e., row/column 1 -10 = the first events of the 10 movies, row/column 11 -20 = the second events, etc.). The black squares on the diagonal indicate cross-movie similarity within the first, second, third, and fourth events of the movies. In PMC (left), all first events showed similar patterns regardless of specific movies, and this tendency decreased in later events further away from between-movie boundaries. EVC (right) showed relatively weaker pattern similarity across movies within the first events compared to PMC. Movie scene images in a were created by the author H. L. using Adobe Illustrator and Adobe Photoshop (adobe.com).
Supplementary Figure 9. Univariate activation. a & b. Whole-brain t-statistic maps showing the brain regions whose activation scale with semantic centrality during movie watching (a) and recall (b). c & d. Whole-brain t-statistic maps showing the brain regions whose activation scale with causal centrality during movie watching (c) and recall (d). All maps were liberally thresholded at p < .001 (Two-tailed one-sample t-tests against zero, uncorrected).   Figure 12. Representational similarity analysis using movie watching phase data and recall transcripts. a. Brain regions that show positive correlations between the movie watching phase cross-event intersubject pattern similarity matrix and the movie annotation sentence embedding vector similarity matrix. b. Brain regions that show positive correlations between the recall phase cross-event intersubject pattern similarity matrix and the recall transcript sentence embedding vector similarity matrix. The recall transcript similarity matrix was first generated within each participant by computing the cosine similarity between the USE vectors of the participant's recall of movie events. The participant-specific similarity matrices were then averaged across participants. In both a and b, representational similarity (i.e., fMRI-text correlation averaged across movies and participants) for each parcel was tested for statistical significance against zero using a randomization test (one-tailed). All maps were thresholded at q < .05 (FDR-corrected across parcels).

Supplementary
Supplementary Figure 13. Effects of semantic centrality on event-specific intersubject pattern correlation including all events. a & b. Intersubject pattern correlation (pISC) for High vs. Low semantic centrality events and the difference (Diff) between the two conditions during movie watching (a) and recall (b) in the posterior medial cortex (PMC; left panels) and early visual cortex (EVC; right panels). All movie events were included in the analysis. For High and Low semantic centrality conditions, white circles represent individual participants (N = 15). Black diamonds represent the mean across participants within each condition. Error bars show SEM across participants. For the difference between High and Low conditions (Diff), black diamonds show the true participant average, and histograms show the null distribution of the mean difference. Two-tailed randomization tests were performed to test whether the differences between High vs. Low semantic centrality conditions were significantly different from zero. Higher semantic centrality was associated with higher pISC in PMC (p = .039) but lower pISC in EVC (p = .008) during recall. No significant relationship between semantic centrality and pISC was observed during movie watching (ps > .05). Thus, the results were qualitatively identical to those obtained after excluding the first events from movie watching data and after excluding the events recalled by fewer than five participants from recall data. *p < .05, **p < .01. Source data are provided as a Source Data file.